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Results 


e OCP boot time: 8 minutes -> 17 seconds 
o 1.6. 32x speedup 
o This is to a shell prompt in Linux 


OCP -> DHCP -> wget -> kexec: 20 seconds 
All userland written in Go 

Linux performance and reliability in firmware 
Eliminate а/ UEFI/ME post-boot activity 


The problem 

e Linux no longer controls the x86 platform m 

e Between Linux and the hardware are at least 
2 75 kernels 

e [hey are completely proprietary and 
(perhaps not surprisingly) exploit-friendly 

e And the exploits can persist, i.e. be written to 
FLASH, and you can't fix that 


The operating systems 


Code you 

kiaw Ring 3 (User) 

about Ring 0 (Linux) 
Ring -1 (Xen etc.) 


Ring -2 kernel and 75 kernel 
Control all CPU resources. 
Invisible to Ring -1, 0, 3 


SMM 75 kernel. Traps to 
8086 16-bit mode. 


UEFI kernel running in 
64-bit paged mode. 


X86 CPU you know about X86 CPU(s) you don't know about 


What's in ring -2 and ring -3? 


IP stacks (4 and 6) 

File systems 

Drivers (disk, net, USB, mouse) 

Web servers 

Passwords (yours) 

Can reimage your workstation even if it's 
powered off 


Ring -3 OS: ME (Management Engine) 


Full Network manageability 
Regular Network manageability 
Manageability 

Small business technology 
Level Ill manageability 

IntelR Anti-Theft (AT) 

IntelR Capability Licensing 
Service (CLS) 

IntelR Power Sharing 
Technology (MPC) 


ICC Over Clocking 

Protected Audio Video Path 
(PAVP) 

IPV6 

KVM Remote Control (KVM) 
Outbreak Containment Heuristic 
(OCH) 

Virtual LAN (VLAN) 

TLS 

Wireless LAN (WLAN) 


Vassilios Ververis: https://goo.gl/j7Jmx5 


e Great overview of many early ME flaws 

e Summary: just about every part of the ME 
software can be attacked 

e Only some of the bugs get fixed ... 


‘Intel ME exploit’: 50M hits 


e “Wired” headline: “HACK BRIEF: INTEL 
FIXES A CRITICAL BUG THAT LINGERED 
FOR 7 DANG YEARS" 

ө How many is that? One billion systems? 

e Bug was in the built-in web server in the ME 
o Yep: the hidden CPU had a web server 


о That evidently you can't turn off 
o Even though docs said you could 


Ring -2 “/; OS”: System 
Management Mode (SMM) 


Originally used for power management 
No time for full details but ... 
o Vectors to 8086 16-bit mode code 
m 1.е. great place for an attack 
o All kinds of interrupts can go here, e.g. USB 
o Nowadays almost all of these go out again to ACPI 
That said, it's a very nasty bit of code 


Vendors use it as secret way to “value-add” 


Are there SMI exploits? 


e ‘system management interrupt exploit" -- 
630K hits 

ө So, yes. 

e Chipsets guarantee that once SMM is 


installed, can't change it, see it, turn it off 
о SMM “hidden” memory at top 8 MiB of DRAM. 
ө SMM maintains vendor control over ... you 


Ring -2 OS: UEFI 


UEFI runs on the main CPU 
Extremely complex kernel 

Millions of lines of code 

UEFI applications are active after boot 
oecurity model is obscurity 


Are there UEFI exploits? 


e Absolutely 

oince UEFI (and only UEFI) can rewrite itself 
o These exploits can be made persistent 

You might even have UEFI fake the process 
of removing an exploit 

e [he only fix? A shredder 


(Some) UEFI components 


CsmVideo 
Terminal 
SBAHCI 
AHCI 
AhciSmm 
BIOSBLKIO 
IdeSecurity 
IDESMM 
CSMCORE 
HeciSMM 
AINT13 
HECIDXE 
AMITSE 
DpcDxe 


ArpDxe 
SnpDxe 
MnpDxe 
UefiPxeBcDxe 
NetworkStackSetupScreen 
TcpDxe 
Dhcp4Dxe 
Ip4ConfigDxe 
Ip4Dxe 
Mtftp4Dxe 
Udp4Dxe 
Dhcp6Dxe 
Їрбрхе 
Mtftp6Dxe 


Udp6Dxe UsbKbDxe 
IpSecDxe UsbMouseDxe 
UNDI UsbBusDxe 
IsaBusDxe XhciDxe 
IsaloDxe USB/XHCl/etc 
IsaSerialDxe Legacy8259 
DiskloDxe Digital TermometerSensor (sic) 
ScsiBus 

Scsidisk 

GraphicsConsoleDxe 

CgaClassDxe 

SetupBrowser 

EhciDxe 

UhciDxe 


UsbMassStorageDxe 


Summary 


27» hidden OSes in your Intel x86 system 
They have many capabilities 

They have network stacks and web servers 
They implement self-modifying code that can 
persist across power cycles and reinstalls 

e They hide, have bugs, and control Linux 

e Exploits have happened 

e Scared yet? We sure are! 


Сап we fix this mess? 


ө Partially ... 


o Moving to AMD is not a solution, they're closed too 
o Don't believe all you read about Ryzen 


We focus on Intel x86 for now 
Reduce the scope of the 2 72 OSes 
Overall project is called NERF 


Non-Extensible Reduced Firmware 
o Extensibility Considered Harmful 


Non-Extensible Reduce Firmware 


e Make firmware less capable of doing harm 
e Make its actions more visible 


ө Remove all runtime components 
о Well, almost all: the ME is very hard to kill 
o But we took away its web server and IP stack 


ө Remove СЕРІ IP stack and other drivers 
e Remove ME/UEFI self-reflash capability 
e Linux manages flash updates 


NERF components 


De-blobbed ME ROM 

UEFI ROM reduced to its most basic parts 
SMM disabled or vectored to Linux 

Linux kernel 

Userland written in Go (http://u-root.tk) 


Removing the ME 


We dont want ME at all; not an option 
If you remove ME firmware, your node 
o May never work again 

o May not power on (as in OCP nodes) 

o May power on, but will turn off in thirty minutes 


Good news: ME firmware has components 


And most are removable 
o Thanks Trammell Hudson 


Removing most of the ME code 


me_cleaner can remove ME blobs 
https://github.com/corna/me_cleaner 

On minnowmax, 5M of 8M FLASH is ME 
me_cleaner.py reduces it to 300K 
Removes web server, IP stack, pretty much 
all the things you don't want "Ring -3" doing 
e Server (SPS) is not yet solved 


Ме_с!еапег on the minnowmax 


BUP 
KERNEL 
POLICY 


HOSTCOMM 


FPF 
RSA 
ҒТРМ 
ClsPriv 
CLS 
SessMgr 
TDT 


(Uncomp. 
(Uncomp. 
(Uncomp. 
(Uncomp. 


(Uncomp. 


(LZMA 
(LZMA 


(Uncomp. 


(Uncomp. 


(LZMA 
(LZMA 


5) 


5) 


0x045000 
0x05a000 
0x08d000 
0x0a8000 
дхдс0000 
0x0c6000 
Өхдсаеөө 
дхд4адод 
дхдағодо 
дхдевдоо 
0х@+4000 


дхд5а9д00): 
0x08d000) : 
0x0a8000): 
Ox0c0000): 
дхдс6000): 
дхдсс385): 
Ox0dc305): 
Oxedfe00): 
дхде8ддо): 
0x0f3906): 
0x0f9452): 


NOT removed, essential 


removed 
removed 
removed 
removed 
removed 
removed 
removed 
removed 
removed 


removed 


It’s an eye test on OCP ... 


BUP b2c2962872f9efb7fc905c53a56c6e47565406eefe350de7bd5ea52c4c3ef264 plain 
BUP 1a24f58f9b04499cb7dcbd481552944946601484912738cfe6bcb9a1dbfe589f plain 
KERNEL 5b419f959814a4dbda06fdcaba4b84ed1a2488a2acb2de1ca2234807bba6d4fa [MATCH] 
POLICY c84a79ee14d7231bd8e967fc8660228bb4f5d75a6c516247d1435cf5d266f46F [MATCH] 
HOSTCOMM  5e54d9f081aecb39571183ea7b6b34e5209e9ed14252457cbf751019932ea92f [MATCH] 
ICCMOD ee1a0bb460d2ea9c7e1669e85a54701c50f33013ae4c10f8dbc25fadddf82bfb [MATCH] 
BASEEXT 84074ba8ba4b6dca24e086be3748c768468b63e18d1ac5909ba1f8e1d0544f9b [MATCH] 


SC 5b22b84a4ac67751a55c280ce6b69c2c9d6d649a97 10031db43a14f39e4a337d [MATCH] 

NM ba2ff3a68035174080a50da2fffab21c6de16b84838c9f7 159ce3ec99e0c5261 [MATCH] 

DM 91cbb5777bb5c5a3c2776edf15dc35b65b6ebda2ae83d0b7ea03cb080e95ea83 [MATCH] 
BUP 1a24f58f9b04499cb7dcbd481552944946601484912738cfe6bcb9a1dbfe589f plain 


KERNEL 5b419f959814a4dbda06fdcaba4b84ed1a2488a2acb2de1ca2234807bba6d4fa [MATCH] 
POLICY c84a79ee14d7231b886e9671:8660228bb4f5d75a6c516247d1435cf5d266f46f [MATCH] 
HOSTCOMM  5e54d9f081aecb3957f183ea7b6b34e5209e9ed14252457cbf751019932ea92f [MATCH] 
ICCMOD ee1a0bb460d2ea9c7e1669e85a854701c50f33013ae4c10f8dbc25fadddf82bfb [MATCH] 
BASEEXT 84074ba8ba4b6dca24e086be3748c768468b63e18d1ac5909ba1f8e1d0544f9b [MATCH] 
SC 5b22b84a4ac67751a55c280ce6b69c2c9d6d649a97 10031db43a14f39e4a337d [MATCH] 
NM ba2ff3a68035174080a50da2fffab21c6de16b84838c9f7159ce3ec99e0c5261 [MATCH] 
DM 91cbb5777bb5c5a3c2776edf15dc35b65b6ebda2ae83d0b7ea03cb080e95ea83 [MATCH] 


Ring -2: Dealing with SMM 


e We have experimental work that directs 
SMM interrupts to kernel handler 

e Requires that kernel run before SMM is 
installed 

e Or that SMM never be installed 

Most preferred: kill SMM 

e Second: vector SMI# to kernel 


Ring -2: On to СЕРІ ... 


e [here's a huge amount of capability іп UEFI 
o Le.a great place to put exploits 

e Some interrupts still go there 
o SECDED 

e We want to remove those opportunities 

e Unified Extensible Firmware Interface 
o Becomes NON-extensible 


(Some) UEFI components 


CsmVideo 
Terminal 
SBAHCI 
AHCI 
AhciSmm 
BIOSBLKIO 
IdeSecurity 
IDESMM 
CSMCORE 
HeciSMM 
AINT13 
HECIDXE 
AMITSE 
DpcDxe 


ArpDxe 
SnpDxe 
MnpDxe 
UefiPxeBcDxe 
NetworkStackSetupScreen 
TcpDxe 
Dhcp4Dxe 
Ip4ConfigDxe 
Ip4Dxe 
Mtftp4Dxe 
Udp4Dxe 
Dhcp6Dxe 
Їрбрхе 
Mtftp6Dxe 


Udp6Dxe UsbKbDxe 
IpSecDxe UsbMouseDxe 
UNDI UsbBusDxe 
IsaBusDxe XhciDxe 
IsaloDxe USB/XHCl/etc 
IsaSerialDxe Legacy8259 
DiskloDxe Digital TermometerSensor 
ScsiBus 

Scsidisk 

GraphicsConsoleDxe 

CgaClassDxe 

SetupBrowser 

EhciDxe 

UhciDxe 


UsbMassStorageDxe 


UEFI Components 


EFI Boot Services Table DXE Services EFI System Table 
DXE Foundation / DXE Dispatcher 


Security Metronome воз Runtime 
Architectural Architectural Architectural Architectural 
Protocol Protocol Protocol Protocol 


Watchdog 
Timer 
Architectural 
Protocol 


CPU Timer 
Architectural Architectural 
Protocol Protocol 


EFI Runtime Services Table 


Real Time 
Clock 
Architectural 
Protocol 


Standard UEFI boot steps 


Platform Initialization (РІ) Boot Phases 


| , UEFI 
i Pre Н B i Interface 


| OS-present 
Ч App, a.k.a. 
| exploit home 


Boot 
- Manager 


Intrinsic B B 
Services : Loader d nvironmen 


Power on [.. Platform initialization . . | [aee 40S БОО] Shutdown 


© 


Step 1: Replace boot with Linux 


B Processor ЕШ 


river 


ES We allow DXE dispatcher if 


E US АСР! and some device 
= "| TE Wnitilization require it. We 
Р ower оп [.. Platform initialization . . | remove most DXEs. We 


© kexec next kernel. 


Step 2: rebuild one part of UEFI 
Note: only limited source available! 


B Processor ЕШ 


Intrinsic 


6 Note that all of SEC/PEI is 
ЕТЕТІ binary, but DxeCore апа 
some small number of DXEs 
can be rebuilt from source. 


Rebuilding bits of EFI 


Security re river Executior 
nitialization nvironment 
(SEC) | tnitiatizati EE 


(РЕ!) 


Ipower оп Г. . Platform initialization 


ө https://github.com/osresearch/he ds/tree/nerf 
e Part of Trammell Hudson's “HEADS” work 


e Allows you to build NERF images with your 
own kernel and initramfs 

e Has shown good results on several servers 
e We are making changes to build with u-root 
e [his all changed just a few days ago ... 


Using Linux makes firmware easier! 


single kernel works on several boards 
We used to finely tune kernel for boards 
o No longer needed 


Caveat: it /s tied to the BIOS vendor 
o Because of ACPI setup 
o Steps for AMI, TianoCore differ 


What about user space? 


Userspace іп Go: u-root (u-root.tk) 
source-based root file system 


ө 5.9M firmware-based initramfs that includes 
o All command source 

o All required Go compiler and package source 

о Go toolchain 

Commands compiled on first use or at boot 
About 200ms to build; 1 ms to run 

Nice from security angle since source visible 


In some cases we want only binary so ... 


Can build all u-root tools into single 
program for compact initramfs 


e File system: 1 program and many symlinks 
e Use Go abstract syntax tree package to 
rewrite commands as packages 

Compile into one binary (takes 15s) 
Doesnt include source code or toolchain 
Reduces footprint to 2M 

Useful when flash space is small (<5М) 


Implications for startup 


Replace all init scripts with Go program(s) 
Do not need systemd, upstart, scripts 
Custom-built Go binary for init is very fast 
Easier to understand than sea of files 
Note: NiChrome, based on u-root, boots 


Chromebook to x11+browser in 5 seconds 
o See me later if you are interested in NiChrome 


- ч ме 


We'd love to have your help! 


Testing 
Improving Travis tests 


o 
o 
e Porting 
o 
o 


ысы атты оо” 
7 ] Ч 7, "257 e 


Contributing 
Documenting 


Extra slides for u-root 


Outline 


Go in 60 seconds 

What u-root is 

How it all works 

Using Go ast package to transform Go 
Where we're going 


Go іп 60 seconds 


e New language from Google, released 2009 
e Creators include Ken, Rob, Russ, Griesemer 
e Not Object Oriented 

o By design, not ignorance 
e Designed for systems programming tasks 

o And really good at that 
e My main user-mode language since 2010 
e Addictive 


Go in 60 seconds:goo.gl/dlJrY G 


// You can edit this code! 
// Click here and start typing. 


package main ө Емегу file has a 
package 
шы e Must import 
var a struct { packages you use 
i,j int e Declare ‘a’ as an 


anon struct 


Go іп 60 seconds 


Could also say: e Note declarations 
are Pascal-style, 
not C style! 

e “Тһе type syntax 
for C is essentially 


} unparsable." - Rob 
var ab Pike 


type b struct { 
l, j int 


Go in 60 seconds: goo.gl/dlJrY G 


e init() is run before 


func init() { | 
ai-2 main 
| e You can have 
many init() 
f | . 
Giza ш | functions 
fmt.Printf("a is %v, bis %v\n", a,b) 9 b is declared and 
} set 


e "v figures out type 


Could also Say ... 


fmt.Printf(“%d", b) 


Package example hittps://goo.gl/X2SqyZ 


// You can edit this code! 


| ».  variables/functions 

// Click here and start typing. : : 

242221 starting In lowercase are 
not visible outside 
package; those starting in 

internal int U ercase are 
Exported int Pp | 
) e No export/public keyword 


var ( 


Package: https://goo.gl/X2SqyZ 


func youCanNotCallFromOutside() { 
fmt.Println("hi") 

) 

func YouCanCallFromOutside() 4 
fmt.Println("hi") 


j 


First class functions:goo.gl/pPAFcJ 


package main 
import "fmt" 


var c = func(s string) (fmt.Println("hi", s)} 
func main() 4 

p := fmt.Println 

p("Hello, E") 


c(" there") 


Easy concurrency: 
https://goo.gl/8BQt8WK 


var done = 
make(chan int) func main() { 
func x(i int) { go x(5) 
fmt.Printf(" od", i) «-done 
done «- 0 ) 


Со іп 60 seconds 


ө Compiler is really fast (originally based on 
Plan 9 C toolchain) 

e V 1.2 was fastest; currently at 1.9, rewritten 
in Go, is still quite fast 

ө Compile all of u-root, including external 
packages, in under 15 seconds 

e Package syntax makes finding all imports 
easy 


u-root 


ө Go-based rootfs 
o Commands/packages written in Go 
o |n one mode, MAX, compiled on demand 
e 1 or 4 pre-built binaries: 
o /init 
о Go toolchain -- if compiling on demand 
ө Туре a command, e.g. rush (shell) 
o rush and its packages are compiled to /ubin and run 
o Compilation is minimal and fast (> second) 


Key idea: $РАТН drives actions 


ө PATH=/bin:/ubin:/buildbin 

о /bin is usually empty 

о /ubin is initially empty 
e /buildbin has symlinks to an installcommand 
e First time you type rush: found in /buildbin 

о Symlink in /buildbin: rush -> installcommand 

o |nstallcommand runs, builds argv[0] into /ubin 

m Execs /ubin/rush 

e Next time you type rush, you run /ubin/rush 


Installcommand is built on boot 


e Init builds installcommand in /buildbin 

e For each d in 
/src/github.com/u-root/u-root/cmds/", init 
creates /buildbin/d -> 
/buildbin/installcommand 


ө init forks and execs rush 
o which may be compiled by the installer and run 


ө init: 206 lines 


“U” is for “Universal” 


ө Single root device for all Go targets 
e New architecture requires only 4 binaries 
e For multi-architecture root, proper 


(rearrangement of paths is needed 
o E.g., /init -> /linux_<arch>/init 


Variations on u-root for embedded 


Not everyone wants source in FLASH 
Some FLASH parts are small 
Hence the root image can take many forms 


But source code never changes 
o Le.no specialized source code for embedded 


Variations of u-root 


More than 4 binaries per architecture: Post-boot model where faster boot is 
some/all commands precompiled, required 

dynamic compilation, multiple 

architectures in one root image 


dynamic compilation, one architecture installed in firmware or local device 


All commands built into one binary which | Usually firmware but also netboot of 
forks and execs each time “Кехес” image 


4 binaries, all commands in source form, Pre- or Post- boot model: u-root В 


А deeper look at u-root “МАХ” 


Standard kernel 


four Go binaries per architecture* 

о init/build binary (part of u-root, written in Go) 
m Merged-in minimized go build tool 

o Compile, asm, link 

All required Go package source 

u-root source for basic commands 


in 5.9M (compressed of course! :-) 


Root structure at boot 
Only one required ПОЭТ 
d it b ед Ё 
urtze — dre 


github.com/ U-root source 


Other source 


go/ 


Init builds directories, mounts, ... 


Init tasks 


/ubin is empty, mount tmpfs on it 

/buildbin is initialized by init with symlinks to 
a binary which builds commands in /bin 
PATH=/go/bin:/bin:/ubin:/buildbin 

create /dev, /proc, /etc 

Create inodes in /dev 

mount procfs 

Create minimal /etc/resolv.conv 


Running first sh (rush) 


e Init forks and execs rush 

e |f rush is not in /ubin, falls to /buildbin/rush 
(symlink->installcommand) runs 

e /buildbin/installcommand directs go to build 
rush, and then execs /ubin/rush 

e And you have a shell prompt 

e From rush, same flow for other programs 


Using Go to write more Со 


e For scripting 

e For dynamically creating shells with builtins 

e For creating small memory pre-compiled 
versions of u-root (“busybox mode") 


Script for ip link command 


fun 1 10082, s: ёс. ОТг аса 
fOr p wv = range 1їїйсег | 
GOOES, %- V«AGGrs (0 
fmt.Printf("$v has Sv", v, addrs) 


ү ] 


ө Result: 


ip: (1 1500 lo up|loopback} has [127.0.0.1/8 ::1/128] 
ip: {5 1500 ethO fa:42:2c:d4:0e:01 up|broadcast} has [172.17.0.2/16 fe80::f842:2cff:fed4:e01/64] 


e But it’s not really a program ... how's that work? 


‘Run’ command rewrites fragment 
and uses the go import package 


e run reads the program 
o If the first char is {', assumes it is a fragment and 
wraps ‘package main’ and ‘func main() boiler plate 
e Import uses the Go Abstract Syntax Tree 


(ast) package: 

o Parses a program 

o Finds package usage 

o Inserts go "import" statements 


The result 


ө run program builds and 
runs the code 
e Uses Go to write new Go 


package main 
import “net” 


import “fmt” 


func main() { 


for Cubieboard 


о Allwinner A10 --> not very fast 
e Wanted to compile all u-root programs into 


one program 


Taking rewriting further 


e With the ast package, we can rewrite 
programs as packages, e.g. 15.00 


package main package Is 


var x = flag.String(‘l’, ...) erri var x = flag.String('ls.l", ...) 
func Init() (...) 


func init() (...) 
func main() { func Main() ( 


e Combine all of u-root into one program 
e Turning 65 programs into one: 10 seconds 


What ts all this good for? 


e Building safer startup environments 

ө \/\/е сап verify the root file system as in 
ChromeOS, which means we verify the 
compiler and source, so we know what we re 
running 

e Much easier embedded root 

ө Security that comes from source-based root 

e Knowing how things work 


But І want bash! 


e |ts ok!: tinycorelinux.net has it 

e The tcz command installs tinycore packages 
ө tcz [-h host] [-p port] [-a arch] [-v version] 

o Defaults to tinycore repo, port 8080, x86 64, 5.1 
Type, e.g., tcz bash 

Will fetch bash and all its dependencies 
Once done, you type 

/usr/local/bin/bash (can be in persistent disk) 


Where to get it 


github.com/u-root/u-root 
Instructions on 
U-root.tk 


Status 


e Demonstrated on 4 motherboards 

e Hope to have a single Go tool to do the job 
in a few months 

e Looking for collaborators 

e While we prefer coreboot-based systems we 
can use u-root оп UEFI-based systems via 
NERF 


Basic builtin(s) 


IS pop 3mm ZA 
hz. СУИС. РЕССОР e 
there ‘{fmt.Println(“there”) y” 


e Create a new shell with hi and there 
commands 


Builtins combine script and rebuild 


package main e This is the ‘cd’ builtin 

— e lives in /src/sh 

import "os" e When sh is bullt, it is 

SEE extended with this builtin 
шаа ө Create custom shells with 


} тээ 
built-ins that are Go code 
func cd(cmd string, s []string) error 4 


Siak ga e eg. temporarily create | 
return errors.New("usage: са one-path") purpose-built shell for Init 

} р Gea 

e etea ө Eliminates init boiler-plate 

return err scripts 


Customize the shell іп a few steps 


e create a unique tempdir 

e copy shell source to it 

e convert sets of Go fragments to the form in 
previous slide 

e Create private name space with new /ubin 

ө mount --bind the tempdir over 
Isrc/cmds/rush/ and runs /ubin/rush 

e You now have a new shell with a new builtin 


Тһе new shell 


ө Child shells will get the builtin 
o since they inherit the private name space 
ө Shells outside the private name space wont 
see the new shell 
e When first shell and kids exit, builtin is gone 
e Custom builtins are far more efficient 


o Need a special purpose shell many times? 
o You can pay the cost once, not once per exec 


